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Memorandum b1k- 8028 


TO : Director, OTR DATE: 20 July 1973 
STAT 
SUBJECT: Program Performance Measurement. 


The Emerging Role of OT2 in establishing 
Requirements for Training. 


I am submitting this memorandum intended as a discussion of 


(1) how to provide the data base necessary to tell OTR how + 


we 
the programs are meeting their objectives on a current ke 


Op 
jf 


(2) an approach to determine to what extent the knowledge end/or 
skills imparted are applied on the job and s 


¢ (3) how OTR can establish requirements as contrasted with 
| historically reacting to requirenents,. 


A concept of a data base is presented which is intended to aggresate 


all of this and also attached is a basic form that can be used to 
respond to all of these goals. 

In order to institute the best type of (program) performance 
measurement a number of aspects must be considered which not only influence 
the behavior and norale of those being measured but also affect the ebility 
to satisfy the student -- our customer, 

This paper is divided into four sections 

Section 1 - 4 Discussion of Program Performance Measurement 
Section 2 - OTR's Role in Establishing Reauirements 
Section 3 ~ Some Observations 


Section 4 - Appendix A-C 


STAT 


airman, bunetions Course 


eee Buy U.S, Savings Bonds Regularly on the Payroll Savings Plan 
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A Discussion of Program Performance Measurement 


There are four steps involved in Program Performance Measurement 
that must be followed ina continually repetitive sequence to achieve 
the full benefits of such efforts. 

The first step requires the careful specification and analysis of 
basic program objectives in each major area of activity. To accomplish 
this one must back away from the particular program being carried on, 
look at their objectives, and ask what are we really trying to 
accomplish. This definition of goals to be accomplished should start 
ideally at the top of the “organization” so that each level can be 
certain that their definition of goals falls within the scope of what 
has been defined by the next higher level. OTR certainly must 
provide the training necessary for agency personnel to accomplish or 
improve the acconplishment of the various agency missions but such 
definition is too broad and must be narrowed. At the other extreme 
for OTR to say we shall train "x" number of people or increase our 
student through-put by some quantity is too narrow and must be 
broadened. Specification of OTR objectives must fall between these 
two extremes. The more we learn about how to reach an objective, 
the more clearly we understand the objective resulting in a constant 
interaction between the decision process and our knowledge of our 
true objectives. 

The second step involves the analysis of the output of a given 
program in terms of the objectives initially specified in the first 
step. For the OTR programs, our output addresses not how many or 
how many more students did we turn out but rather what improvements, 


‘ 
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| knowledge and/or new skills have we provided to the student. Will 
our educational efforts influence his behavior on the job and improve 
his job performance. This is often referred to as feedback a term 
taken from engineering servomechanism theory. 

the third step cails for measurement of the total costs of the 
program ~- not just for one year but over at least several years 
ahead. This would require OTR to identify the resources expended 
in each program, to accumulate resource costs by program and to 
extrapolate into the future periods from a historical cost data 
base. 

The fourth step involves the analysis of alternatives if and 
only if programs are competing for limited resources. As an 
expository application let us examine how these steps or this cycle 
of events applies to the Information Science Program and the 
Information Science for Intelligence Functions Course within that 
program. 

Course objectives serve two purposes: 

1. They express the desired results uf our customers in 
terms of accomplishments or goals to be achieved by 
the course. 

2. They provide the basis or elements fundamental to course 
evaluation. | 

The most difficult aspect of any professional performance 
measurement. scheme is not how to measure but what should be measured. 
What should be measured is dependent upon how one is organized and 
whether accountability and responsibility for activities which 
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satisfy customers needs are clearly defined and understood by those 


charged with such responsibility and accountability. Clearly 


organizational analysis is beyond the scope of this paper nevertheless 
the interrelationship between the design of an organization and an 
evaluation of an organization's performance must be clearly established. 
These ideas have been used in our Information Science Program. 
For example I have asswned from our previous planning that the single 
overall objective required by step one of our Information Science 
Training Program at the Chief/Information Science Training Staff level 
is to educate Intelligence Professionals in the Information Science 
Disciplines. For the Current FY 73 Community Program segment, the 


numbers of students planned were as follows: 


2 offerings -- Survey -- 3 weeks -— 30 stdts = 60 

2 offerings -- Functions -- 4 weeks -— 25 stdts = 50 
2 offerings ~— Inf. Sci. -- 1 week -— 25 stdts = 5 

Total Students 160 

Total Course Weeks 16 


Assuming for step two that the proper output measure of the 
Information Science Program objective is to quote numbers of students 
exposed to Information Science then I could certify for the Functions 
Course that 27 + 30 or 57 students completed the Functions Course 
during FY 73. One can further document the specific inputs (and 
their costs required in step three) required to generate this output 
of 57 students i.e., lecture time of resident and guest faculty, 
course preparation time, computer time and charges (by problem and 


lecture if necessary) etc. From a Systems Analysis point of view, 


. 


ey 
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the output (57 students) is tied to the related and required inputs 
(the resources and their costs) but is this a sufficient and proper. 
measure? This measure of output as a single measure reflects the 
number of students put through the course but does not reflect the 
full range of values or benefits provided to our students and their 
respective organizations. In order to measure the value provided 
by the Functions Course and its contribution to the Information 
Science Program overall, it is necessary to go beyond the single 
measure of student output and resource input to determine whether 
we accomplished the course objectives and accordingly satisfied 
the students our customers. What is missing is the careful specification 
and analysis of the program objectives which produces a more meaningful 
output measure. | 

To elaborate further -- for the Functions Course alone -- four 
priority ranked specific student-oriented objectives were established 
(step one): 


1. To familiarize you with the terminology and basic 
techniques of Information Science. 


2. To develop your capability of identifying and defining 
problems in your professional intelligence field which 
are amenable to solution by information science techniques 
and to solve such problems at the elementary level. 


3. To improve your conmunications capabilities in conferring 
with information science professionals. 


4. To encourage you to pursue further the development of 
your own, and your organization's information science 
resources and capabilities. 


How well we accomplish these objectives in the Functions Course 


with a fixed set of resource inputs is reflected in the course 
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. evaluations completed by the students. A summary of the student 
evaluation responses from the last Functions Course demonstrates 
to what degree these objectives were fulfilled for that particular 
class (step two): 


STUDENT EVALUATION-FINDINGS 


OBJECTIVE OUTSTANDING / EXCELLENT GOOD FAIR 
1, 27 Students 3 0 
90% of the class 10% -— 
2. 15 13 2 

50% 43% 7% 
3. 21 7 2 

70% 23% 7% 
4. 23 6 1 

77% 20% 3% 


From these findings one can conclude that all objectives were 
accomplished. Since objectives 2 and 3 rank the lowest, efforts 
should be directed toward improving those areas for the next 
offering of the Functions Course. A follow-up questionnaire, four 
to six months after the course, would further confirm the degree of 
job application accomplished by the students of each course. 

These objectives and the associated evaluation scheme used for 
the Functions Course may not be applicable for all courses conducted 
by OTR. Each course must have sreeteica ied oat set of objectives 
based on the unique requirements of the customer needs to be satisfied 
by each course. How each course is conducted depends upon (1) the 
subject material to be presented, (2) the skill and knowledge of the 
assigned manpower presenting the course and (3) any procedural 


directives issued by top and/or middle management that apply 
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specifically to a particular program or course. These are the 
factors that influence not only the specification of objectives 
but also the degree to which the course objectives can be 


accomplished. 


OTR's Role in Establishing Requirements 

Although this process, its steps and their interrelationship 
have been explained using the Functions Course as a "real live 
intelligence application" the same concepts apply to any level within 
the organization. For example, the objectives that OTR must accomplish 
have in the past depended upon the requirements that have been 
levied by "Top" management. How OTR is organized has depended upon 
(1) the entire list of such requirements and (2) the aggregation of 
similiar activities and/or courses (requirements) into manageable 
(school) segments. Such aggregations must provide for clearly defined 
areas of responsibility and authority which are the pre-requisites to 
accountability. To say the least I have been disappointed in the limited 
agency career development program that exists only for specific segments 
or very narrow specialties. The time has come for OTR to become more 
ageressive not only in career development training but also in much 
of the currently named “component” training. The training that OTR 
conducts for the Directorate of Operations and the Directorate of 
Intelligence I consider to be basically component training. In order 
to effect batter utilization of training assets by OTR, OTR must 


effectively plan, organize and control the total training effort of 


ey ee 


Approved For Release 2003/04/29 : CIA-RDP83-00058R000100120007-6 


Approved For Release 2003/04/29 : CIA-RDP83-00058%4000100120007-6 


the agency. Planning sets the stage, organizing sets the resources 
and controlling sets the degree to which the organization will 
continue to survive. Planning determines a Schedule of what is to 
be accomplished, Organizing determines the resources to be invested 
and how to accomplish the planned schedule and controlling determines 
how well the plan was accomplished. Since control requires a 
measured comparison of what was accomplished to what was planned to 
be accomplished, there can be no control without a succinct plan. 
The degree or amount of control required is dependent upon the 
professional capabilities of the personnel. In this regard OTR must 
decide what role it is to play in Agency Training... This really means 
a restatement of the objectives to be accomplished, time-phased ina 
reasonable balanc.2 between assets, resources and time for accomplishment. 
I am reminded of the little boy who was asked by his father (a world 
reknowned Product:.on Control expert) what he wanted for his birthday 
next month. The toy replied "a baby brother." His dad responded 
with "that's impossible son." To which the boy quickly replied 
"you have taught ne all of the principles and practices of effective 
production control, just put more men on the job". 

Once the objectives are carefully specified and analyzed, then 
OTR can determine if the present organization is properly structured 
to accomplish this priority-ranked list of objectives. The only 
definitive principles of organization structure that apply are that 
(1) each organization falls somewhere between the extremes of being 


functionally organized and being product or service organized and 


‘ 
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(2) like-activities should be grouped into manageable segments so 
that the specified objectives of such "grouped" activities can be 
responsibly accomplished. 

If OTR is to exert more influence in career development, then 
OTR must write career development plans (programs) in conjunction 
with each responsible directorate. These career development prograns 
then become the focal point of the OTR program planning and the 
subsequent course planning. The existence of career development 
plans does not require OTR to be organized along the same career 
structure. In fact, an OTR element such as the Information Science 
Program can cut across a number of career development plans by 
providing segments of instruction or courses that fit into various 
career progression patterns. Other such examplas include the IWA 
introductory orientation, the mid-career and management courses and 
the senior seminar. 

If OTR recognizes a need or void in trainirg then a course or 
program should be developed and presented. Demand for such a progran 
can be generated through effective advertising campaigns. We should 
look to the methods used in successful marketing research and advertising 
campaigns and apply such tactics to researching the need and promoting 
newly developed programs. 

To determine how well each course is meeting its objectives, I 
offer as an example (appendix A) the form used in the Functions Course. 
As a follow-up questionnaire, I offer the form (appendix B) designed fer 
the Functions Course. To determine the allocation of manpower/time 
assets, 1 offer the Faculty/Staff Activity Report (appendix C) as the 
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basic time allocation data collection device. One can thus relate 
resource expenditures to the benefits provided to our customers -- 


the students and to those responsible for review of career development. 


Some Observattons 

It is often charged that such measurement sets up biases in 
decision-making by concentrating on kests and ignoring intangibles 
and human factors which cannot be quantified. Or conversely by 
naively attempting to put numbers on such imponderable elements 
thereby misleading the decision-maker. Such thinking often forces 
personnel to play games. -For example (in the evaluation of personnel 
who (1) present lectures or (2) participate as guest lecturers in 
other courses.) If I knew that my performance was to be evaluated 
on (1) the number of students who were lectured (2) the number of 
my lecture hours and (3) the ratio of students per lecture hours 
then I would make certain that my lecture hours per course were at 
a maximum and that I lectured only to large student-groups or classes. 
I could so maximize my performance measures that I would always be 
number one on this performance roster. 

However sincere these critics may be -~ they reflect a complete 
misunderstanding cf the relevent issue. And sometimes they simply 
reflect the chagrin that particular pet projects may not show up 
well under such measurement schemes. 

Program Performance measurement does require a systematic analysis 
(the means) of program proposals and decisions, concentrating on those 


particular decisions (the ends) which have inherent budgetary 
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consequences, Please note that systematic analysis does not have to 
be quantitative and is not co-extensive with quantitative analysis. 
The word "analyze" does not have the same meaning as the words 
"quantify" or "measure" although analysis often includes some form 

of measurement. Management by Objectives, the PPB process or whatever 
the current "in" title of a good management philosopy is, all of these 
concepts seek to subject to a systematic analysis both the tangible 
and intangible clements of a program decision. We live in a world 
that must make decisions often using limited or meagre information. 
This is more akin to the European Style of Management which opts 

for a decision as contrasted with the American Style which opts for 
"adequate" information before making the decision. In any event 

let us not becoma so Management by Objectives (MBO) oriented that 

we become manager: who can't make decisions unless the plan calls 


for a decision. 
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THNFORMATICN SCIENCE FOR INTELLIGENCE FUNCTIONS 


LASS: 


INTRODUCTION. Tha Staff is concerned with tha quality of its educational 
program. Your constructive comments are solicited and will be used as part 
of the basis for improving the ability of future presentations of the 
Functions Ccurse to meet the information scieace training requirements of 
user organizations 


1. Waat is your overall impression of this course? 


Outstanding Excellent. Good Fair Pocr 


e. Did we fulfill each of our course objectives for you? Indicate below: 


(a) To familiarize you with the terminology and basic technigues 
of Information Science. 
! 
Outstanding Excellent___ Good Fair Poor 
dentifying and defining problems 
co field whica area amenable to 
tecliniques and to solve’: sucn 


(b) To davelon your capability of id 
in your professional intel ligenc 
solution by iaformetion science 
problems at uhe elementary level. 


Puts tanding Exceilent Good Fair Poor 


(c) To improve your communtceticns capabilities in conferring with 
information science profassionals. 


tstanding Excellent___-_ Good Fair Poor 
(d) To encourage you to pursue further the development us your own, 
and your organization's information science resources and 
capabilitias. 


Outstanding Excellent Good Fair Poor 


(e) Do you believe the course content is compatible with the course 
objectives? Yas Ro 
Discuss this compatibility and describe ay changes that you 
think would increase this compatibility 
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Would you recommend this course to a co-worker? 
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Waat is your job title and what are your major tasks? 


What are your personal reason(s) for attending this course? 


Raflecting on your learning from the course what percent of the course 
do you feal will contributa or be of use to you? Please identify by . 
entering one (x) check for each line. 


a. Immediate Use 


25% 50% 19% iGOZ 
25% SUb 13% 100% 


b. Long-range Use 


Please rate the course in light of your answers to Questions #4, #5, 
#6 and #7. 


Outstanding Excellent Good Fair Poor 


9. 


As a follow-up to your end-of-ccourse evaluation, any subsequent 
comments based on work exparience at your home station would be most 
welcome and helpful in updating course relevance, ould you be 
interested in accomplishing a rnost~-graduate quasticnnaire four months 
after completion cf this Caurse? 


Yes No 


eee ete 


Approved For Release 2003/04/29 : CIA-RDP83-00058R000100120007-6 


Approved For RgJease 2003/04/29 : CIA-RDP83-00058RQ90100120007-6 


APPENDIX - B 


ry 


Approved For Release 2003/04/29 : CIA-RDP83-00058R000100120007-6 


3. Reflect on your learning from the course this week. For each subject element listed please indicate 


your personal reaction to the APP riate 


Approved 


SUBJECT ELEMENT I can see no 


application for 
this element in 
my work, 


Elementary System 
Concepts 


Basic Programming 


Statistics 

Library Programs 
DELPHI 

Decision Trees 
Network Analysis 
Intelligence Problems - 


making use of 
these new techniques 


ro 
or Release 


Iam aware of 
a few appli- 
cations for this 
element. 


Iam aware 
of many 
applications 
for this 
element in 
my work, 
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I'd like to incor- 
porate the infor- 
mation & tech- 
niques from this 
element in my 


work. 
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Iwill certainly 
incorporate 
the infor- 
mation and 
techniques 
from this 
element in my 
work, 


£ x 1 d 
3. Reflect on your leaz RB RroveaF Sr RASS POSTS? Ela. Aes VIdssRoIDTOOIsdoOTS Te” Please indicate 
your personal reaction to Aa @ appropriate space 


SUBJECT ELEMENT I can see no Iam aware of Iam aware I'd like to incor-| Iwill certainly, 
application for a few appli- of many porate the infor-~| incorporate : 
this element in cations for this | applications | mation & tech- the infor- 
my work. element. for this niques from this | mation and 

element in element in my techniques 
my work. WOTK, from this 


element in 


my work. oe 


Library Programs 


ence Problems - 


Le 
LE 


making use of these 
new techniques 
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acer aementnanntesaniearer nn sett sen intent elma eens 


SUBJECT ELEMENT I can see no Iam aware of Iam aware I'd like to incor-|] I will certainl! 


application for a few appli- of many porate the infor- |. incorporate 

this element in cations for this | applications | mation & tech- “the infor- 

my work, element, for this niques from this | mation and 
elernent in element in my : techniques 
my work, work, . "| from this 


element in my 
worl, 


inforrnation Storage And 
Retrieval 


Queueing 


Basic Programming 


wecision Theory 


Library Programs 


intelligence Problems - 


making use of these 
ew techniques 
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UBIECT ELEMENT I can see no Iam aware of 
application for a few appli- 
PE PP 
this element in cations for this 


my work, 


Retrieval Operations 


Tiumnan Factors 


Intelligence Problems - 


vnaking use of these 
dev techniques 


element. 


0042 


Iam aware 
of many 
applications 
for this 
element in 
my work, 


I'd like to incor- 
porate the infor- 
mation & tech- 
niques from this 

element in my 
work, 
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the infor- 
{ 
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pinta nt ce as: le 


I will certainly 
incorporate 


mation and 
techniques 
from this 
element in 
my work, 
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NAME: 


This form is simple to maintain if done each day. 


amount of time spent performing any activity listed. 


WEEK OF: TO 
Indicate the 


(Indicate 


others if necessary.) An activity is considered significant if 


it requires more than 15 minutes of your time to do it. 


In the 


case of short duration jobs, i.e., filing, merely record the 


approximate total time spent in that activity during the day. 


turn in to branch chief by Friday noon each week. 


ADMINISTRATION: 
Management: 


letter/report-research 
letter/report~preparation 
letter/report-coordinating 


Briefings-preparing 
Briefings~presenting 
Briefings~attending 


Meetings-preparation 
Meetings-attcnding 


Telephone 


Clerical: 


typing 
filing 
reproduction 


meetings 
telephone 


training 


mime ene ermenee Pe mr ae mare are oa am el te 2 SS Se Re RP Se ne Sn MR a ns er we st we 


CURRICULUM: 
Instruction: 


Research/study 
Lesson preparation 
Lesson presentation 
Lesson attendance 
Coordinating 
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CURRICULUM ACTIVITY CONTD. 


Scheduling: 


preparation 
coordinating 


Student Advising/Evaluating 


Faculty Advisor Time 
Seminar Evaluation 
Paper/report evaluation 
Briefing evaluation 
Counseling 

Record administration 


Exercises: 
Development 
Preparation for 
Participation 


ee ce ee ee ne an ek a Hh SS AE ON RN AE KM SR SW ee 


Professional development 


Proficiency maintenance 


ee ee cate ee ee Se A HH RY te a SN ET NS ENN UR Nm Ee 


Travel/TDY 


Leave 
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